Performance Level Descriptors Research Articles

Setting standards for a diagnostic test of aviation English for student pilots

Standard-setting is an essential component of test development, supporting the meaningfulness and appropriate interpretation of test scores. However, in the high-stakes testing environment of aviation, standard-setting studies are underexplored. To address this gap, we document two stages in the standard-setting procedures for the Overseas Flight Training Preparation Test (OFTPT), a diagnostic English test for ab initio pilots, aligned to the International Civil Aviation Organization (ICAO)’s Language Proficiency Rating Scale (LPRS). Performance-level descriptors (PLDs) were empirically generated in Stage 1 in collaboration with six subject matter experts (SMEs). These PLDs made explicit the correspondence between linguistic performance levels within the target language use domain and the ICAO scale. Findings suggest that the ICAO scale is not fine-grained enough to distinguish levels of linguistic readiness among ab initio pilots, nor does it adequately reflect the knowledge, skills, and abilities valued by SMEs within this domain. In Stage 2, 12 SMEs were recruited to set standards and were divided into two groups to investigate the replicability of Ebel method standard-setting procedures. Cut scores were determined for the OFTPT reading and listening tests, which were inferentially linked to the LPRS. There were no significant differences in the cut scores arrived at by both groups and reliability was excellent, suggesting that test users can have confidence in the standards set.

Early detection measures: addressing Unemployability using Modified Angoff in situational judgment test

This is the first standard setting study in Malaysia aimed at developing Performance-Level Descriptors (PLDs) and determining cut scores that differentiate undergraduates with low, moderate and high employability skills using a newly developed Situational judgment Test (SJT). SJT items from five domains critical for future job employment: 1) critical thinking and problem solving, 2) enterprising, 3) integrity, 4) leadership and 5) teamwork were developed. Modified Angoff method involving PLDs and cut scores were developed by five subject-matter experts. Two test instruments were used: 10 practice and 36 original bilingual Malay-English SJT items. The PLDs and two cut scores to distinguish low, moderate and high employability skills were validly developed, without compromising consequential validity. The findings benefit undergraduates with low employability as they can be trained with current global job readiness skills by the universities, which also forms the practice implication of this study. Another practical implication is academicians can be further encouraged to understand the theoretical framework of establishing standard setting for employability.

THEORETICAL FRAMEWORK FOR TEACHERS SELF- ASSESSMENT TO TEACH 21ST CENTURY SKILLS

The context of educational reforms taking place in many countries gives special relevance, as the curriculum enters the 21st century skills or so-called transversal skills. It is necessary to ascertain how successful the teaching of these skills is already so that the school management together with teachers develop evidence-based or data-driven professional development solutions. The aim of the research is to create a theoretical framework and corresponding self-assessment tools, how the teacher can assess for himself to what extent he has succeeded in achieving the competence to teach students in the 21st century skills required to achieve the goals set by the school in the context of the ongoing education reform in the country. For the self-assessment of teachers’ competencies, performance appraisal is used, comparing to what extent does teaching in a particular case correspond to good practice using performance level descriptors as tools - a set of teachers' learning progressions and tests. The developed set of teachers' professional learning progressions is based on data-based and field-tested analytical descriptions of performance levels from expert work. It includes 17 progressions developed within the framework of categories and criteria, the practical testing of which in school practice is planned in the authors' further research.

Generating Performance‐Level Descriptors under a Principled Assessment Design Paradigm: An Example for Assessments under the Next‐Generation Science Standards

AbstractThe educational testing landscape is changing in many significant ways as evidence‐based, principled assessment design (PAD) approaches are formally adopted. This article discusses the challenges and presents some score scale‐ and task‐focused strategies for developing useful performance‐level descriptors (PLDs) under a PAD approach. Details of the strategies are illustrated using an example based on the Next‐Generation Science Standards (NGSS).

The Preparation of Stewards with the Mastery Rubric for Stewardship: Re-Envisioning the Formation of Scholars and Practitioners

A steward of the discipline was originally defined as “someone who will creatively generate new knowledge, critically conserve valuable and useful ideas, and responsibly transform those understandings through writing, teaching, and application”. This construct was articulated to support and strengthen doctoral education. The purpose of this paper is to expand the construct of stewardship so that it can be applied to both scholars and non-academic practitioners, and can be initiated earlier than doctoral education. To accomplish and justify this, we describe a general developmental trajectory supporting cross-curriculum teaching for stewardship of a discipline as well as of a profession. We argue that the most important features of stewardship, comprising the public trust for the future of their discipline or profession, are obtainable by all practitioners, and are not limited to those who have completed doctoral training. The developmental trajectory is defined using the Mastery Rubric construct, which requires articulating the knowledge, skills, and abilities (KSAs) to be targeted with a curriculum; recognizable stages of performance of these KSAs; and performance level descriptors of each KSA at each stage. Concrete KSAs of stewardship that can be taught and practiced throughout the career (professional or scholarly) were derived directly from the original definition. We used the European guild structure’s stages of Novice, Apprentice, Journeyman, and Master for the trajectory, and through a consensus-based standard setting exercise, created performance level descriptors featuring development of Bloom’s taxonometric cognitive abilities (see Appendix A) for each KSA. Together, these create the Mastery Rubric for Stewardship (MR-S). The MR-S articulates how stewardly behavior can be cultivated and documented for individuals in any disciplinary curriculum, whether research-intensive (preparing “scholars”) or professional (preparing members of a profession or more generally for the work force). We qualitatively assess the validity of the MR-S by examining its applicability to, and concordance with professional practice standards in three diverse disciplinary examples: (1) History; (2) Statistics and Data Science; and (3) Neurosciences. These domains differ dramatically in terms of content and methodologies, but students in each discipline could either continue on to doctoral training and scholarship, or utilize doctoral or pre-doctoral training in other professions. The MR-S is highly aligned with the practice standards of all three of these domains, suggesting that stewardship can be meaningfully cultivated and utilized by those working in or outside of academia, supporting the initiation of stewardship prior to doctoral training and for all students, not only those who will earn PhDs or be scholars first and foremost. The MR-S can be used for curriculum development or revision in order to purposefully promote stewardship at all levels of higher education and beyond. The MR-S renders features of professional stewardship accessible to all practitioners, enabling formal and informal, as well as self-directed, development and refinement of a professional identity.

The Mastery Rubric for Bioinformatics: A tool to support design and evaluation of career-spanning education and training.

As the life sciences have become more data intensive, the pressure to incorporate the requisite training into life-science education and training programs has increased. To facilitate curriculum development, various sets of (bio)informatics competencies have been articulated; however, these have proved difficult to implement in practice. Addressing this issue, we have created a curriculum-design and -evaluation tool to support the development of specific Knowledge, Skills and Abilities (KSAs) that reflect the scientific method and promote both bioinformatics practice and the achievement of competencies. Twelve KSAs were extracted via formal analysis, and stages along a developmental trajectory, from uninitiated student to independent practitioner, were identified. Demonstration of each KSA by a performer at each stage was initially described (Performance Level Descriptors, PLDs), evaluated, and revised at an international workshop. This work was subsequently extended and further refined to yield the Mastery Rubric for Bioinformatics (MR-Bi). The MR-Bi was validated by demonstrating alignment between the KSAs and competencies, and its consistency with principles of adult learning. The MR-Bi tool provides a formal framework to support curriculum building, training, and self-directed learning. It prioritizes the development of independence and scientific reasoning, and is structured to allow individuals (regardless of career stage, disciplinary background, or skill level) to locate themselves within the framework. The KSAs and their PLDs promote scientific problem formulation and problem solving, lending the MR-Bi durability and flexibility. With its explicit developmental trajectory, the tool can be used by developing or practicing scientists to direct their (and their team's) acquisition of new, or to deepen existing, bioinformatics KSAs. The MR-Bi is a tool that can contribute to the cultivation of a next generation of bioinformaticians who are able to design reproducible and rigorous research, and to critically analyze results from their own, and others', work.

A developmental trajectory supporting the evaluation and achievement of competencies: Articulating the Mastery Rubric for the nurse practitioner (MR-NP) program curriculum.

BackgroundAdvanced practice registered nursing (APRN) competencies exist, but there is no structure supporting the operationalization of the competencies by APRN educators. The development of a Mastery Rubric (MR) for APRNs provides a developmental trajectory that supports educational institutions, educators, students, and APRNs. A MR describes the explicit knowledge, skills, and abilities as performed by the individual moving from novice (student) through graduation and into the APRN career.MethodA curriculum development tool, the Mastery Rubric (MR), was created to structure the curriculum and career of the nurse practitioner (NP), the MR-NP. Cognitive task analysis (CTA) yielded the first of the three required elements for any MR: a list of knowledge, skills, and abilities (KSAs) to be established through the curriculum. The European guild structure and Bloom’s taxonomy of cognitive behaviors provided the second element of the MR, the specific developmental stages that are relevant for the curriculum. The Body of Work method of standard setting was used to create the third required element of the MR, performance level descriptors (PLDs) for each KSA at each of these stages. Although the CTA was informed by the competencies, it was still necessary to formally assess the alignment of competencies with the resulting KSAs; this was achieved via Degrees of Freedom Analysis (DoFA). Validity evidence was obtained from this Analysis and from the DoFA of the KSAs’ alignment with principles of andragogy, and with learning outcomes assessment criteria. These analyses are the first time the national competencies for the NP have been evaluated in this manner.ResultsCTA of the 43 NP Competencies led to seven KSAs that support a developmental trajectory for instruction and documenting achievement towards independent performance on the competencies. The Competencies were objectively evaluable for the first time since their publication due to the psychometric validity attributes of the PLD-derived developmental trajectory. Three qualitatively distinct performance levels for the independent practitioner make the previously implicit developmental requirements of the competencies explicit for the first time.DiscussionThe MR-NP provides the first articulated and observable developmental trajectory for the NP competencies, during and beyond the formal curriculum. A focus on psychometric validity was brought to bear on how learners would demonstrate their development, and ultimately their achievement, of the competencies. The MR-NP goes beyond the competencies with trajectories and PLDs that can engage both learner and instructor in this developmental process throughout the career.

An Example of Empirical and Model Based Methods for Performance Descriptors: English Proficiency Test

Great emphasis is given to the development of high-stake tests all around the world and in Turkey. However, limited emphasis is given to adequate score reporting. Too much emphasis on rankings and almost no emphasis on performance level descriptors (meaning of the scores) have leaded a “ranking culture” in Turkey. There is an immense need to raise awareness about score reporting and performance level descriptions in Turkey. This study aims to raise awareness about the use of performance level descriptors in a high-stake exam in Turkey, an English proficiency exam. The study sample is consisted of 630 undergraduate students who took the 2016-2017 English proficiency exam of a public university in the southwest of the Turkey. In order to identify the potential exemplars, two types of item mapping methods (i.e. experimental based method and model-based method) were used in the present study. Item grouping for performance level descriptors provided hierarchical and interpretable structure. Using these performance level descriptors, it is possible to give criterion referenced feedback to each student about his/her reading abilities.

DESIGNING & PILOTING ONLINE TESTS AS PART OF A TEACHER COMPETENCE ASSESSMENT

Approaches for competence assessment are becoming increasingly important to plan school development goals, especially during complex educational changes. This paper outlines authors’ experience and recommendations from the designing and piloting of teacher online tests as part of a wider competence management process. A developed theoretical model of a universal teacher competence profile is presented. Tests represent four thematic parts related to teacher performance in classroom to facilitate: student cognitive activation, student self-regulation, student collaboration, and leveraging digital. Test questions were developed based on a framework of teacher performance assessment that consists of a structured set of performance level descriptors to help determine teacher level of competence (0-4) according to criteria developed. Online tests were completed by 197 teachers. Test results were analysed to determine their validity. Proposed online tests can be used for interpreting assessment results of teacher knowledge and beliefs necessary for teaching 21st century skills and plan teacher professional development for goal attainment in schools. 

A standards-based approach for reporting assessment results in South Africa

This article proposes the use of a standards-based approach to reporting results from large-scale assessment surveys in South Africa. The use of this approach is intended to address the key shortcomings observed in the current reporting framework prescribed in the national curriculum documents. Using the Angoff method and data from the Annual National Assessments, the article highlights how standard setting procedures should be conducted to develop meaningful reports that provide users with relevant information that can be effectively used to identify and develop appropriate interventions to address learning gaps. The findings of the study produced policy definitions and performance level descriptors that are proposed for use in enhancing the reporting of results for grade six English and mathematics. Moreover, the findings also indicate that the reporting of the Annual National Assessments using the national curriculum reporting categories overestimates the percentage of learners classified at the lowest performance levels and underestimates those in the next category. This finding has serious implications for the implementation of targeted interventions aimed at improving learning for all. The paper concludes by noting areas of further research for enhancing the use of results of large-scale assessment surveys and for supporting schools and teachers in addressing specific learning needs of all learners, especially the poor and marginalised.

Enhancing the Interpretability of the Overall Results of an International Test of English-Language Proficiency

The purpose of this study was to enhance the meaning of the scores of an English-language test by developing performance levels and descriptors for reporting overall test performance. The levels and descriptors were intended to accompany the total scale scores of TOEFL Junior® Standard, an international test of English as a second/foreign language. The study addressed two issues: the number of performance levels that could be meaningfully reported and the information that should be included in the performance level descriptors. Data from 3,607 students who took an operational test form were used. Although our methodology built to some extent on earlier work (Papageorgiou et al., 2015), we demonstrate how content and construct differences between the tests of each study dictated use of different types of data in order to construct meaningful performance descriptors and select the cut-offs for the levels. We emphasize the importance of establishing the psychometric quality of the reported overall levels and the empirical basis for the content of their performance descriptors, and we discuss the implications of our work for similar efforts in the future.

Writing Performance Level Descriptors and Setting Performance Standards for Assessments of Modified Achievement Standards: The Role of Innovation and Importance of Following Conventional Practice

Alternate assessments of modified academic achievement standards (AA-MAS) must be designed, developed, implemented, and validated following the same rigorous principles and procedures used for other assessments. However, the uniqueness and unfamiliarity of the target population for these assessments requires innovative thinking, especially in regard to writing relevant performance level descriptors (PLDs) and setting appropriate achievement standards. In this article we discuss considerations for (a) likely designs of AA-MAS; (b) the writing of PLDs that are relevant to the target examinee population for these assessments; and (c) standard setting methods that would be appropriate for these assessment designs, as well as special considerations relevant to those standard setting methods.

A Guide to Understanding and Developing Performance‐Level Descriptors

There has been much discussion recently about why the percentage of students scoring Proficient or above varies as much as it does on state assessments across the country. However, most of these discussions center on the leniency or rigor of the cut score. Yet, the cut score is developed in a standard‐setting process that depends heavily on the definition for each level of performance. Good performance‐level descriptors (PLDs) can be the foundation of an assessment program, driving everything from item development to cut scores to reporting. PLDs should be written using a multistep process. First, policymakers determine the number and names of the levels. Next, they develop policy definitions specifying the level of rigor intended by each level, regardless of the grade or subject to which it is applied. Finally, content experts and education leaders should supplement these policy definitions with specific statements related to the content standards for each assessment. This article describes a process for developing PLDs, contrasts that with current state practice, and discusses the implication for interpreting the word “proficient,” which is the keystone of No Child Left Behind.

Integrated, Comprehensive Alignment as a Foundation for Measuring Student Progress

This paper describes a comprehensive model of alignment that provides a foundation for meaningful reporting of students' academic progress over time. The model includes both horizontal and vertical alignment as integral parts of the development of content standards, test blueprints, items, item pools, instruments, performance level descriptors, and performance standards. It also explains why comprehensively integrating alignment into development processes results in procedural efficiencies and gains in validity evidence for the measurement of student progress. Cautions against reducing focus on content in favor of alignment are given. All parts of the comprehensive alignment model described in this paper either have been or soon will be implemented in a large‐scale assessment program.

Aligning a State's Alternative Standards to Regular Core Content Standards in Reading and Mathematics

The No Child Left Behind Act of 2001 allows states to assess students with “significant cognitive disabilities” on “alternative content standards” for determining adequate yearly progress. Alternative standards that align with regular content standards can allow for a continuum of performance expectations from very basic to approaching grade level, therefore tapping into the abilities of students with significant cognitive disabilities. Performance level descriptors were developed in an effort to identify students' level of attainment for each content standard. States must demonstrate that assessments align with articulated alternative standards and that alternative standards align with the state's content standards. The purpose of this study is to examine a methodology for aligning alternative content standards to regular content standards.